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This research report presents the results of a STEM summer program on robotics and game design. 
The program was part of a three-year study funded by the National Science Foundation. Children in 
grades four through six participated in a two-week summer camp in 2015 to learn STEM by 
engaging in LEGO® EV3 robotics and computer-based games using Scalable Game Design. Twenty- 
eight students participated in the study that took place in a small urban community in the Rocky 
Mountain West. This paper reports on the results of this part of the study, specifically, how children’s 
computational thinking skills developed and how their self-efficacy in technology, attitude toward 
engineering and technology, and 21" century skills changed as a result of their participation. 
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According to the Bureau of Labor Statistics, U.S. Department of Labor (2016), the need for 
mathematicians, computer systems analysts, and biomedical engineers is projected to grow from 21- 
23% by 2024. Preparing students to succeed in mathematics and technology is crucial to ensuring 
access to these and other STEM occupations of the future. Moreover, the number of engineering 
students is not increasing and in some instances is declining while the demand for engineers is 
expected to continue to grow (Hirsch, Carpinelli, Kimmel, Rockland, & Bloom, 2007). One reason 
that students are not choosing to study engineering is lack of information about the field, what it 
entails, and what engineers do (Hirsch et al., 2007). Exposing underrepresented students to pre- 
engineering skills through robotics and game design has the potential to increase their interest and to 
provide them with the skills needed to broaden participation to create a diverse STEM workforce 
(National Research Council (NRC), 2011). 

To address this need in STEM education, a student- and teacher-focused project funded by the 
National Science Foundation was implemented in Wyoming. The goals of the three-year quasi- 
experimental study were to examine how spatial reasoning, computational thinking, children’s self- 
efficacy in technology, and attitudes toward STEM and STEM careers changed as a result of 
participation and how well teachers implemented the STEM program. Specifically, we provided 
students with access to LEGO® EV3 robots, its accompanying software MINDSTORMS8®, and 
Scalable Game Design software and protocols (Repenning, Webb, & Ioannidou, 2010; Webb, 
Repenning, & Koh, 2012). Teachers received training to deliver the instruction through an online 
professional development course. 

The purpose of the two-week summer program was to field test the double effect of teaching both 
gaming and robotics to improve students’ spatial visualization and computational thinking skills. 
Robotics and game design have not only been extolled for their role in learning but have also been 
identified as pathways to broaden participation in STEM and STEM-related careers (Caron, 2010; 
Sheridan, Clark, & Williams, 2013). We implemented the summer program to inform Year 3 
deliverables. In Year 1, we piloted the instruments and the iterative project model. In Year 2, 
teachers taught robotics in the fall and game design in the spring to measure the effect of a single 
treatment. The research questions that guided this part of the study were as follows: 
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1. How did students’ self-efficacy in technology, attitudes toward engineering and technology, 
and 21° century skills change as a result of engagement in robotics and game design? 

2. How did children’s computational thinking (CT) compare and contrast on Maze Craze, 
Frogger, and Pac Man games? 

3. What learning preferences and STEM interests did students report during focus group 
interviews? 


Theoretical Framework 

The framework that undergirds this study is Learning-for-Use (LfU) (Edelson, 2001). LfU is a 
technology design framework that is based on four principles: (a) knowledge construction is 
incremental in nature, (b) learning is goal directed, (c) knowledge is situated, and (d) procedural 
knowledge needs to support knowledge construction (Edelson, 2001). These principles inform 
robotics applications and game design and lend themselves to the interventions implemented in this 
study. 

The first and fourth principles of the LfU model are the incremental development of new 
knowledge and procedures. The goal behind the progression of two intervention components— 
robotics and game design— is to engage students in an incremental process. This idea allows 
students to incrementally add new concepts to memory, while subdividing existing concepts or 
making new connections between concepts. Procedural strategies for supporting and reinforcing 
incremental learning include observation, discussion, reflection, and application. New knowledge 
informs and empowers students to become proactive in their own learning. In its second and third 
principles, LfU recognizes that acquisition of knowledge is goal directed and situated. The realization 
of gaps in one’s knowledge, perhaps as the result of an elicited curiosity or external demand, can be 
used as a motivational goal for acquiring new knowledge. The intervention was designed to 
encourage goal-directed tasks as students created computer games to learn and apply spatial 
reasoning and computational thinking skills, which are needed for computer science and information 
and communications technology (ICT) careers. 


Literature Review 
The bodies of literature that support this study are robotics and digital gaming. The literature that 
support this study is presented below. 


Robotics 

Robotics programs have resulted in an increase in students’ comfort level with applications of 
STEM, development of 21“ century skills, and increased interest in pursuing STEM-related programs 
beyond high school (Brand, Collver, & Kasarda, 2008). LEGO® robotics, specifically, is widely used 
in K-8 settings as an authentic and kinesthetic way to improve children’s problem-solving skills, 
reinforce science applications and concepts, and build upon informal learning activities often done at 
home (Karp & Maloney, 2013). Informal STEM experiences in robotics extend students’ experiences 
into alternative spaces where there is greater potential for learning to be “...self-motivated, voluntary, 
and guided by the learner’s needs and interests” (Dierking, Falk, Rennie, Anderson, & Ellenbogen, 
2003, p. 109). In this study, robotics provided students with opportunities to engage in authentic 
learning, inquiry, and scientific processes such as observing and recording data, evaluating and 
providing feedback, and using evidence to revise thinking, planning, and actions. 


Digital Gaming 

Digital game playing has also been used successfully to teach mathematics problem solving 
(Chang, Wu, Weng, & Sung, 2012) and can be used as a social practice to support the development 
of “strategic thinking, planning, communication, application of numbers, negotiating skills, group 
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decision-making, and data-handling” (Li, 2010, p. 429). Studies revealed that children showed 
considerable improvement in regard to developing positive attitudes toward learning mathematics 
through gaming (Kebritchi, Hirumi, & Bai, 2010). Using digital games in mathematics classrooms 
has led to favorable attitudes toward learning mathematics and to increases in mathematics 
achievement and student success. Several studies explore the use of games to improve student 
learning and computational thinking (Li, 2010; Repenning et al., 2010; Webb et al., 2012). For 
example, Scalable Game Design (SGD) project consists of instructional units to support game 
computational thinking through the use of game designs such as AgentSheets and AgentCubes 
(Repenning et al., 2010). 


Methodology 


Participants and Setting 

Twenty-eight students (23 males; 5 females; 4 underrepresented minorities; and 4 students with 
disabilities) were recruited into the program. The two-week summer camp was held in August 2015 
at the Starbase facility (Starbase is a Department of Defense program that seeks to motivate fifth- 
grade students to explore STEM through inquiry-based curriculum) in Cheyenne from 9 a.m. — 3 
p.m., Monday — Friday with morning and afternoon breaks and a 1-hour lunch and recess. We ran 
two concurrent classes—one on robotics and the other on game design—with 14 students each. 
Students switched classes in the afternoon to receive four hours of instruction on robotics and game 
design each day. 

The main instructor for the robotics class was an engineer at Starbase. The principal investigator 
was the main instructor for the game design class. Lesson plans in the robotics course focused on 
using MINDSTORMS® to engage in basic programming to incorporate ultrasonic sensors, color 
sensors, and touch sensors. Students used LEGO® EV3 robotic kits while working in pairs to make 
the basic car, two- and three-wheeled rovers, and a sumo bot that were programmed to traverse a 
color-coded map, race on a simulated track, or bot fight in a ring, respectively. Lesson plans in the 
game design class focused on guiding students to create Maze Craze, Frogger, and Pac Man games 
using Scalable Game Design. 


Data Analyses and Data Sources 

Mixed methods were used to analyze quantitative and qualitative data in this phase of the study. 
The Self-Efficacy in Technology and Science instrument (SETS) developed by Ketelhut (2010) was 
used in this study. Self-efficacy as defined by Bandura (1977) is the belief that one can successfully 
perform specific tasks. We administered three of the SETS subscales to measure students’ self- 
efficacy in technology: videogaming (8 items), computer gaming (5 items), and using the computer 
to solve problems (5 items). Cronbach alpha reliability ratings ranged from 0.79 to 0.93, which were 
in the acceptable range. The Student Attitudes toward STEM survey developed by the Friday 
Institute (2012) was modified to include two subscales: engineering and technology (9 items) and 
21“ century skills (11 items). Cronbach alpha coefficients were in the acceptable range (a > 0.83). A 
5-point Likert scale that ranged from 1| (strongly disagree) to 5 (strongly agree) was used to rate 
items on the SETS and the Student Attitudes toward STEM surveys. The 7-statistic was used to 
analyze pre-post scores on these surveys. The confidence interval was .95, and statistical significance 
was established at a = .05. 

Computational thinking (CT) was analyzed using a rubric developed by the principal investigator 
to rate students’ games. A three-point rubric was established with an interrater reliability of 86%. 
Scores on the rubric ranged from 1 for emerging, 2 for moderate, and 3 for substantive evidence of 
CT. Focus group data were collected the final week of the summer camp during an interview with 
several randomly selected students. These data were analyzed using the constant comparative method 
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(Strauss & Corbin, 1990) to identify emergent themes and patterns related to the participants’ 
learning and interest in STEM. Finally, two members of the research team collected field notes and 
conducted classroom observations. 


Results 


Self-Efficacy, Engineering/Technology Attitude, and 21 Century Skills 

Twenty-one students completed the pre-post surveys. The data (see Table 1) reveal no significant 
differences on the SETS for computer gaming and computer use. However, there was a significant 
decline on the videogaming subscale: (¢ = 2.126; p = 0.046; Cohen’s d = 0.477). Cohen’s d shows a 
moderate effect size for this decline. While pre-post scores increased on Attitudes toward 
Engineering and Technology (M = pre: 3.87 (Std. Dev.: 0.80); post: 3.93 (Std. Dev.: 0.72)) and 21" 
Century Skills (WV = pre: 4.12 (Std. Dev.: 0.72); post: 4.24 (Std. Dev.: 0.65)), results of a paired f-test 
show no statistically significant differences on either subscale. 


Table 1: Results of Self-Efficacy in Technology Scale by Cohort 
Construct (n=21) Pre-Survey Standard Post-Survey Standard 


Mean Deviation Deviation 
Videogaming 4.32 Ra 4.20* 0.65 
Computer gaming 4.08 0.76 4.04 0.68 
Using the Computer 4.05 0.99 4.04 0.80 
* p< 0.05 
Computational Thinking 


Descriptive statistics were also used to rate students’ games for computational thinking using a 3- 
point rubric. Students were taught how to design three different types of games: Maze Craze, 
Frogger, and Pac Man. To analyze students’ games, we evaluated the students’ code, worksheets, and 
game functionality using the six International Society for Technology in Education (ISTE) definition 
for CT (i.e., problem formulation; abstraction; logical thinking; algorithms; analysis and 
implementation; and generalization and transfer). The majority of students completed the Pac Man 
game, which was the most difficult of the three games. 

The results of analyzing the Pac Man game show that four students exhibited emerging CT 
strategies (threshold from | — 1.5), nine students exhibited moderate CT strategies (threshold from 
1.6 to 2.5), and five exhibited substantive CT strategies (threshold from 2.6 to 3.0). Thus, the 
majority of students exhibited moderate or substantial CT strategies on one of the most difficult 
games. To further understand CT on all of the games, we completed additional analysis on four focal 
students (all names pseudonyms) using the same rubric described above. They were randomly 
selected from among eight students who completed all three of the games. The descriptive data are 
shown in Figure 1. 
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Figure 1. Focal Students’ Computational Thinking Scores by Game Type. 


Interestingly, Kate and Latrice, who were underrepresented females in a class that was 
predominantly male, showed greater levels of computational thinking than their male counterparts on 
two of the games. Latrice, who was also a student with disabilities, had the highest average CT score. 
When these data are analyzed by game type, the Pac Man game had the highest average CT rating (M 
= 2.00) compared to Maze Craze (M=1.92) and Frogger (M=1.92). Overall, mean scores support the 
finding that the CT scores of students in the summer cohort fell into the moderate range. Screenshots 
of each type of game are shown in Figures 2-4. 
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Figure 2. Maze Craze Screenshot (Kate). 
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Figure 3. Frogger Screenshot (Latrice). 
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Figure 4. Pac Man Screenshot (Devon). 


Qualitative data included excerpts from a questionnaire during focus group interviews with seven 
randomly selected student participants. The focus group interview with two girls and five boys was 
transcribed verbatim. Portions of the transcribed interview are presented below: 


Interviewer: What one or two things did you think were cool about participating in robotics? 

Student 1: What I thought was cool about...robotics was that we got to build our own robots and 
try out different things with our sensors, and we got to activate our robots.... 

Student 2: | liked building the robots, and I also liked programming the robots and finding out 
how far you could make the thing go. 

Student 3: I liked it so much I have been to Starbase three times...two camps and once at school. 

Student 4: I like building robots and putting on the sensors. 

Interviewer: What one or two things did you think were not so cool about robotics? 

Student 1: What I did not think was so cool about the robots was that we had to do certain steps, 
like we struggled through, and it was hard. I think they should make it easier. 

Student 4: They should try to bring in people that have jobs with robots to help us out. That 
would be helpful. 

Interviewer: How many of you are doing robotics for the first time? Oh, one. How many for the 
second time? Three. And how many are pros? What! Oh, all of the rest of you. 

Interviewer: What one or two things did you think was cool about gaming? 

Student 3: What I liked about the gaming was that we learned how to make our own avatar thing 
out the little pixels, and programming was fun, too. 

Student 4: I get to create my own game. Just creativity, and you can actually play the game. 

Student 6: | like what we are doing right now with Pac Man. My favorite game was my first 
game. I made snakes, and it was really fun. I think snakes are cool. 

Student 7: | like how you can make it impossible for people to beat your game and that the 
teacher told me she never tried [a game like mine] before. 

Student 2: | don’t really like doing gaming. I like putting things together. It was really hard for 
me. The computer did not work very well. This is the first time [doing gaming]. 

Interviewer: How many of you are doing gaming for the first time. (Hands raised.) Oh, all of you 
are doing this for the first time. 

Student 7: Gaming was not as easy as robotics. It’s hard to follow the instructions. You can 
easily get confused and can do something wrong.... 


Themes and patterns were found in the qualitative data. The first theme—building robots— 
emerged among three of the seven students. Two students mentioned enjoying the sensors. A second 
theme that emerged from the transcribed data was creating or programming robots or games. Three 
students mentioned creating or programming explicitly, while two others were implicit in their 
references to creativity: “7 made snakes, and it was really fun; I like how you can make it impossible 
to beat your game.” The third emergent theme was the level of difficulty involved in programming 
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either robots or designing games. Three students stated “it was hard” and that specific steps had to be 
followed to make the robot move or to create games. The final theme that emerged was prior 
knowledge and experience with robots. However, none of the students had prior experience with 
game design. The novelty and high learning curve for developing computer games influenced 
students’ attitudes toward videogaming and computer gaming, even though most of the students 
mentioned they enjoyed playing several off-shelf computer games (i.e., Mario, Minecraft, etc.) 
before participating in the study. While only one student mentioned the importance of having STEM 
professionals as guest speakers, this was a good suggestion in terms of helping students understand 
different aspects of engineering and technology and could improve their aspirations to pursue a 
STEM career. 


Discussion 

The results of this study reveal a great deal about student efficacy, computational thinking, and 
student interest in robotics and game design. The data suggest these children self-selected the 
summer camp because they had a high interest in robotics, gaming, or both. Their pre-scores on the 
surveys were high, indicating that results may have been impacted by a ceiling effect. While non- 
significant, STEM attitude scores in engineering and technology increased slightly along with 21" 
century skills. The highest score on the SETS survey was videogaming, but the scores declined 
significantly from pretest to posttest. Focus group data suggest some reasons for the decline, 
including a preference for robotics over game design, the difficulty in following directions to make 
the game, and making games that were impossible to win. One of the survey questions on the 
videogaming subscale specifically addressed students’ efficacy as it related to winning a game. Most 
of the students’ scores declined on this item after participating in the program. While the sample is 
small, we noticed that some girls were active gamers, and one female student with Asperger’s 
Syndrome enjoyed game design and benefited from the step-by-step process. Moreover, an 
examination of her code revealed a departure from the SGD protocol to include her own nuances. 
Her game (see Figure 3 above) was judged as one of the most creative by her peers during a 
showcase on the final day of the camp. 

In terms of developing computational thinking, focus group data suggest CT scores may be 
correlated with interest, enjoyment, and exposure to game design. None of the students had 
participated in game design before the summer program while almost all of them had prior 
experience in robotics. Field notes and classroom observations revealed most of the students in the 
program enjoyed working on robotics and playing each other’s games. Engagement in robotics and 
gaming provides opportunities for students to engage in STEM content and creates pathways to 
STEM careers (Caron, 2010; Sheridan et al., 2013). 


Significance 

The findings presented in this research report informed the project team about the importance of 
combining the treatments of robotics and game design to increase students’ spatial reasoning and 
computational thinking skills prior to implementing the Year 3 study. Instruction in both robotics and 
game design courses should be made more explicit (to be addressed during professional 
development). While minorities, females, and students with disabilities had high engagement, they 
were underrepresented in the program. This is an important finding that needs to be explored further. 
Recruiting females and other underrepresented students has been difficult in informal STEM 
programs (Leonard, J. Buss, A., Gamboa, R., Mitchell, M., Fashola, O. S., Hubert, T., et al., in press; 
Repenning et al., 2010), particularly in rural settings. This study is no exception. Perhaps using intact 
classrooms, as suggested by Webb et al. (2012) is one important mechanism for broadening 
participation. Our work is ongoing and will culminate in 2017. 
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